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ABSTRACT 



This report is the last in a four-part series of 
Strategies for Change. It offers examples for comparing the 
effectiveness of different instructional systems and of instructional 
evaluation in general. The first evaluation, on remedial English 
procedures, compares the effectiveness of instruction in English 1A 
freshman composition in terms of attrition rates. In the next, two 
comparisons of computer-assisted learning serve as examples of 
evaluating different instructional systems in terms of meeting 
objectives as measured by an examination. Finally, a number of 
informal evaluative efforts are reported that deal primarily with the 
solicitation of student opinion on the ef fectiveness of a learning 
situation. (Author/CA) 
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Part I of Strategies for Change examines current educational 
literature and finds that community colleges, as with all educa- 
tional Institutions, have made few efforts at developing aggres- 
sive administrative programs to foster Innovation and equally few 
efforts to compare Instructional systems with respect to effective- 
ness and costs. Part II presents a case history of the Coast 
Community College District's efforts of operating five such admin- 
istrative programs. Part III describes some initial efforts at 
comparing the costs of different instructional systems. This 
part offers some examples of comparing the effectiveness of dif- 
ferent instructional systems and of instructional evaluation in 
general . 

Despite widespread literature regarding innovation and spe- 
cific innovation projects, as found in such magazines as Telecom- 
munications » Media , and Instructional Technology , relatively little 
is reported with respect to evaluating the relative effectiveness 
of any one particular innovative teaching technique as compared 
to any other. Moreover, what evidence is available suggests that 
it really doesn’t make any difference. There is good reason to 
think however, that this evidence can be discounted inasmuch as 
it depends heavily upon normative evaluative data as opposed to 
the summative type of evaluation that measures the program against 
its specified educational objectives. 
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If comparing relative costs of different instructional progrars and 
systems Is difficult, comparing the effectiveness of these systems even 
more so* These pages present examples of evaluative efforts exercised by 
the Coast Community College District in the past several months. The first 
of these deals with remedial English procedures and compares the effectiveness 
of instruction In English 1A Freshman Composition in terms of grades earned 
by students and In terms of attrition rates. This is an example of normative 
evaluation based upon grades assigned students by instructors. 

The next two comparisons serve as examples of summative evaluation of 
different instructional systems In terms of meeting objectives as measured by 
an examination. This kind of comparison avoids problems of subjective evalua- 
tion characterizing normative comparisons and replaces them with equally 
difficult problems of test validity. 

Finally, a number of informal evaluation efforts are reported. These 
deal primarily with solicitations of student opinion with respect to the 
effectiveness of a learning system. If community colleges serve the primary 
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purpose of meeting student needs, this type of evaluative technique is the 
only one that really makes sense, assuming that students have accurate per- 
ceptions of their educational needs. 

REMEDIAL ENGLISH PROCEDURES; BEFORE AND AFTER 

Starting with the Fall semester 1968-69, remedial English at Orange Coast 
College was taught to students requiring it as an integral part of their 
experience in English 1A, Freshman Composition. This procedure represented a 
change ever previous years during which students needing remedial work in 
grammar, syntax and other matters first completed a remedial English course 
before enrolling in Eng ish 1A. 

The Office of Educational Development of the Coast Community College Dis- 
trict gathered data for the purpose of determining, if possible, the extent 
to which student performance in English 1A as measured by grades and attrition 
rates changed substantially af t er the inauguration of the new policy regarding 
remedial work in English. Exhibit IVA shows grade assignments for all students 
enrolled in English 1A during this four-year period. The column headed "Before" 
shows grades earned by students during the four semesters spanning the Fall of 
1966 and the Spring of 1968. The "After 1 column shows grades earned by stu- 
dents during the four semesters starting with the Fall of 1968 and ending with 
the Spring of 1970. The terms "Before" and "After" at the head of these columns 
indicate that the right-most grade distribution reflects grades earned after 
the new procedures were inaugurated. The other column indicates grades 
earned before their inauguration. 

Grade point averages for the English 1A course increased slightly as 
between the "Before" and "After" groups. Statistical number-pushing showed 
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GRADE DISTRIBUTION FOR ENGLISH 1A 





Before 


After 


Grade Point Average 


2.49 


2.67 


Percentage Completing Coarse 


63.2 


65.1 


Percentage Completing Course 
^access fully 


57.4 


62.5 


Number receiving A 


397 


627 


Number receiving B 


1,022 


1 ,493 


Number receiving C 


1,022 


1,128 


Number receiving D 


200 


126 


Number receiving F 


46 


7 


Number receiving Cr 


0 


0 


Number receiving NCr 


0 


19 


Number receiving I 


48 


123 


Nusiber receiving W 


1,481 


1,664 


Number receiving WF 


33 


6 _ 


Total Enrollment 


4,249 


5*193 



EXHIBIT IV A 
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that this increase is significant at the .01 level of confidence (Ferguson, 
pages 155, 167-168). This means that the difference in grade point averages 
between the "Before" group of students and the "After" group of students 
probably did not come about by sheer chance. The percentage of students com- 
pleting the course with a grade of A, B, C, D, F, or Credit increased from 
63.2 for students taking English 1A during the years 1966 through 1968 to 65.1 
for those enrolling in English 1A from 1968 through 1970. Application of 
statistical procedures shows that this increase probably did not come about 
by chance either, that is to say, that the increase is significant statisti- 
cally at the .01 level of confidence. 

The percentage of students completing the course successfully, that is, 
completing it with either an A, B, or C grade rose from 57.4 percent for the 
"Before" group to 62.5 percent for the "After" group. This increase of 5.1 
percentage points is also statistically significant at the .01 level of 
confidence. 

This is not to say, however, that the increase of grade point average or 
of percentage of students completing the course is necessarily an Important 
increase or that it necessarily vindicates the changes in procedure. It does 
say that for some reason a few more students completed the course in the 
"After" group than completed the same course in the "Before" group and that 
students completing the course earned slightly higher grades. Whether or 
not this reason is directly related to the changes in remedial instruction 
procedures cannot be determined from these numbers. 
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COMPUTER ASSISTED LEARNING TO TEACH COMPUTER OPERATIONS 

During the Spring semester, 1969-70, two Orange Coast College instructors 
designed and implemented a number of computer assisted instruction (CAI) seg- 
ments for use in the computer operations course offered at their college. 

This work was done under the auspices of a Faculty Fellowship. Among other 
things, the segments included routines specifically devoted to the teaching 
of IBM System/360 Disk Operating System computer operator commands. 

The procedures followed in this project consisted of six steps. Each of 
these is briefly described in the paragraphs that follow. 



Design CAI Segments 

The objectives of the operator command portion of the computer operations 
course Included the f ollowing : 

a. Students should be able to recognize correct Disk Operator 
Systems (DOS) operator commands in relation to specified 
computer functions to be performed. Achievement of this 
objective Is measured by multiple choice examination ques- 
tions. 

b. Students should be able to identify computer system func- 
tions that take place as a result of executing specified 
operator commands as measured by multiple choice examina- 
tion questions. 

c. Students should be able to distinguish between correct 
and incorrect operator commands and operator procedures 
as measured by true-false examination questions. 

d. Students should be able to provide the correct computer 
operator commands needed to cause specified computer 
functions to be performed as measured by fill-in exam- 
ination questions. 

e. Students should be able to describe specific steps to be 
taken by a computer operator to cause the computer system 
to perform specified computer functions as measured by 
essay examination questions. 
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f . Students should be able to define certain terms used in 
computer operations as measured by essay examination 
questions. 

g . Students should be able to describe and provide examples 
of the operation of specific operator commands as measured 
by essay examination questions. 

Computer assisted learning; segments simulating the disk operating system 
of the IBM System/ 360 Model 50 Computex* and offering tutorial material teach- 
ing operator commands were designed to offer students the opportunity to 
control a complex computer software system without requiring the dedicated 
use of the computer system. Altogether, these computer assisted learning seg- 
ments replace approximately five hours of lecture materials dealing with COS 
operator commands. 

Test the CAI Segments 

The computer assisted instruction segments were tested in two ways. 

First, they were used by students in the computer operations course offered 
during the Spiting semester, 1969-70. Second, they were examined and used by 
ten data processing instructors attending the Summer Institute in Data Pro- 
cessing conducted by Orange Coast College during the months of July and August 
1970. This Institute was attended by experienced data processing instructors 
who came for the purpose of studying advanced topics in modern computer system 
programming and operation. 

The students using the materials during the Spring semester found them to 
be a most enjoyable way to learn computer operations. It was thus established 
that students undergoing training in the subject matter, using computer 
assisted instruction materials, would not react unfavorably to the materials 
by virtue of their intrinsic nature* The data processing teachers similarly 
found the CAI segments most promising and became quite excited about their use 
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This testing program and the resulting changes made to the CAI segments 
assured us that the learning materials that had been developed were acceptable 
on the part of both data processing teachers and data processing students and 
deserved further examination to determine their relative effectiveness in 
achieving specific objectives of the computer operations course. 

Establish Control and Experimental Groups 

The population of this study consists of 28 students who completed the 
computer operations course offered at Orange Coast College during the Fall 
semester, 1970-71. At the start of the course, these students were divided 
into a control group (15 members) and an experimental group (13 members). 

The control group received conventional classroom instruction in matters 
dealing with computer operation. This instruction consisted of classroom 
lectures and demonstrations. The experimental group studied computer operation 
through the use of computer assisted instruction segments. 

The students were divided into control and experimental groups on the 
basis of two factors. First, each student was asked to fill out a short 
questionnaire designed to determine the degree of experience he might have in 
data processing prior to enrolling in the course. The questionnaire also 
determined what other data processing courses the student had completed. Each 
member of the computer operations class also took the SCAT II examination. 

The members of the class were then divided Into two groups, A Group and 
B Group, based upon previous training and experience as determined from the 
class questionnaire. The mean SCAT II scores of these two groups were then 
compared to determine whether or not there was a statistically significant 
difference in measured ability between the groups. Exhibit IV B shows the 
composition of the two groups and the SCAT II scores for each member of the 
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SCAT II SCORES 



GROUP A 



GROUP B 



Student Scat Student Scat 



1 


88 


16 


80 


2 


79 


17 


78 


3 


70 


18 


71 


4 


69 


19 


61 


5 


63 


20 


61 


6 


61 


21 


58 


7 


80 


22 


53 


8 


58 


23 


53 


9 


55 


24 


49 


10 


53 


25 


48 


11 


51 


26 


47 


12 


48 


27 


36 


13 


47 


28 


29 


14 


43 






15 


17 







EXHIBIT IV B 
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class. Homoscedasticity for the two groups with respect to SCAT II scores 
was verified by finding the ratio between their variances and using It to con- 
sult a table of the F distribution. No statistically significant (.05 level 
of confidence) difference between the variances was found. The W. S. Gosset 
formula was used to calculate t (Ferguson, pages 155, 167-168). This 
yielded a t score of 0.29 which was not significant at the .05 level. This 
Indicated that there was no significant difference between the two groups 
with respect tc SCAT II scores. Thus, the control group and the experimental 
group were judged to be about equal, both in terms of ability as measured by 
the SCAT II score and In terms of previous training and experience as deter- 
mined through the use of the class questionnaire. 

Conduct Training 

Instruction was given to the control group following conventional class- 
room procedures Including lecture and classroom discussion and demonstrations. 
The experimental group did not receive lecture or classroom discussion In the 
area of computer operations but limited their training experiences in this 
area to the computer assisted Instruction segments. The control group partic- 
ipated In two classroom sessions of three hours each during which the conven- 
tional instruction was offered. The experimental group spent at least this 
much time proceeding through the CAI segments and were told that they could 
spend as much time as they liked using these materials. Both the experimental 
and the control group members could approach the Instructor and aslc questions 
on an Individual basis. 

In addition to classroom Instruction and computer assisted Instruction, 
students participated In the execution of laboratory exercises In which they 
actually operated the computer system. Each student did this as a member of a 
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team. The class was divided Into four teams, two teams from the experimental 
group and two teams from the control group. Each team was given a number of 
laboratory exercises and made use of the computer facilities to execute the 
exercise. Each laboratory exercise required 45 minutes for execution. 

Administer Examination 

After the training was completed, all students in the class took an 
examination testing for the achievement of the objectives of this portion of 
the course. The examination consisted of 43 multiple choice questions, 20 
true-false questions, 10 fill-in questions, four essay questions, and required 
students to describe the specific computer operations of 13 operator commands 
commonly used by computer operators. 

Compare the Examination Scores of the Control and Experimental Groups 

Exhibit IV C shows the examination scores for students in both Group A 
and Group B. Examination scores were tested for homogeniety of variance as 
described earlier with respect to SCAT II scores. This examination showed 
that there is no statistically significant difference (.05 level of confi- 
dence) between the variances in examination scores for the two groups. Applica- 
tion of the Gosset t test yielded a t score of 0.12. This score reveals no 
significant difference between mean examination scores of students In Groups A 
and B at the .05 level of confidence. 

Results 

Objective analysis following the procedures described above show no 
significant difference In performance between that group of students who 
studied operator commands through the use of CAI segments as compared with 
those who studied it using only conventional classroom Instruction techniques. 
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EXAMINATION SCORES 



GROUP A 




GROUP B 




Student 


Exam 


Student 


Exam 


i 


276 


16 


256 


2 


254 


17 


276 


3 


232 


18 


254 


4 


250 


19 


257 


5 


266 


20 


285 


6 


266 


21 


261 


7 


254 


22 


236 


8 


268 


23 


262 


9 


256 


24 


253 


10 


278 


25 


258 


11 


198 


26 


268 


12 


250 


27 


254 


13 


272 


28 


120 


14 


240 






15 


154 







EXAM SCORE STATISTICS 




EXAM SCORE STATISTICS 




SAMPLE 


15 


SAMPLE 


13 


MAXIMUM 


278 


MAXIMUM 


285 


MINIMUM 


154 


MINIMUM 


120 


RANGE 


124 


RANGE 


165 


MEAN 


247.6 


MEAN 


249.23 


VARIANCE 


1078.97 


VARI /NCE 


1647.36 


STANDARD DEVIATION 


32.85 


STANDARD DEVIATION 


40.59 



EXHIBIT IV C 
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Subjectively, however, two observations were nade by the instructor. First, 
that portion of the computer assisted learning materials in which the opera- 
tion of a computer console was simulated by an APL terminal was not as 
enthusiastically received by the students as was the tutorial computer assisted 
learning materials presented through the APL terminals. The instructor 
explained this phenomenon by reasoning that an APL terminal simulating a con- 
sole typewriter terminal offers nothing more in terms of learning experiences 
than does the coi;sole typewriter itself. Students may as well go directly to 
the computer console and try to learn to operate it by relying entirely upon 
operator's manuals and other manufacuter f s applications. The tutorial mate- 
rials, on the other hand, offered learning experiences that analyzed students' 
errors and guided them through various learning processes. This was received 

i 

much more warmly on the part of students than were the simulation exercises 
and the instructor believes that, for this subject matter at least, the 
tutorial method of computer assisted learning is by far the best. 

The instructor also observed that the experimental group seemed consid- 
erably more confident and were better organized in the laboratory exercises 
than were members of the control group. Teams 1 and 2, the ceams from the 
control group, had considerable difficulty executing the laboratory exercises 
with the computer system. One of them did not complete them within the 
allowed time at all. Teams 3 and 4, however, both drawn from the experimental 
group, completed the exercises within the allotted 45 minutes and Team 3 com- 
pleted it in only 15 minutes. Although not all of this difference in perform- 
ance can be directly attributed to the computer assisted instruction exercises, 
one would like to think that improved performance at the computer itself is 
more important than the lack of score differences on a written examination. 
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The laboratory performance difference Is not merely a result of the experi- 
mental group working with computer machinery and the control group not. In 
point of fact, both the experimental and the control group worked extensively 
with the computer throughout the course. The only additional computer exposure 
enjoyed by the experimental group over the control group had to do with the 
completion of the computer assisted learning segments themselves. This addi- 
tional exposure cannot alone result in the different performance levels 
observed in the computer room. 

Conclusions 

This small study reenforced in many ways what educators have learned 
about other similar studies regarding the relative effectiveness of different 
instructional techniques (Dubin and Taveggia). By and large, such studies all 
demonstrate that, at least judged by teacher evaluation in the form of course 
grades and examination scores, any one instructional technique is neither 
better nor worse than any other. This too, apparently, is true for computer 
assisted instruction with respect to teaching computer operator commands. In 
short, the objective results of this study, namely that there is not signif- 
icant difference in test scores between the control and experimental groups, 
does not tell us anything we did not expect to find out.. However, the sub- 
jective results that students seem more enthusiastic and seem to perform better 
In the real world environment of the computer room, seemed more encouraging. 

Of course, one may argue that the increased enthusiasm on the part of 
students as well as on the part of the data processing instructors exposed to 
the C£I segments is merely the result of novelty. This may be so. However, 
it renders the differences in computer room performance nonetheless important. 
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If the use of novel Instructional techniques will do a better job of teaching, 
then surely we must use them. It remains to be seen whether or not the novel 
aspects. If that is what they are, of computer assisted instruction will soon 
wear off. If they do, then we must look beyond novelty and try to judge the 
merit of this new instructional technique on its intrinsic worth. 

COMPUTER ASSISTED INSTRUCTION FOR LAW ENFORCEMENT 

The analysis described here was performed as part of Project CALCOP, an 
experimental project conducted jointly by Golden West College and the Los 
Angeles Police Academy. It was financed under a grant from the Law Enforce- 
ment Assistance Administration under the Omnibus Crime Bill. 

Project CALCOP served a two-fold purpose. First, the project sought to 
develop a computer assisted learning system for the purpose of training In 
the area of search and seizure. Second, the project evaluated the effective- 
ness of the computer assisted learning system. In doing this, the project 
examined the hypothesis that the learning system designed by the project, con- 
sisting In independent study and CAI exercises, would be more effective than 
conventional classroom instruction. 

Procedures followed In Project CALCOP are enumerated below: 

1. Objectives of training programs In search and seizure and 
rules of evidence were formulated. 

2 . An examination designed to test the degree to which the 
objectives were met was developed and validated. 

3. A syllabus of cognant material to be used for study pur- 
poses on an Independent basis was prepared. 

4. Case problems simulated through the use of the computer 
terminal were prepared and implemented. 

5. Training was conducted using the computer assisted learning 
system and the syllabus at Golden West College. Training 
also took place through conventional classroom instruction 
at the Los Angeles Police Academy. 

16 
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6. The examination was administered to police cadets at both 
the Los Angeles Police Academy and t:he Golden West Academy. 
Performance on this examination was compared between the two 
groups to determine if the computer assisted instruction 
techniques were more or less effective than conventional 
classroom techniques. 

Evaluation of the learning procedures designed as part of Project CALCOP 
followed conventional statistical procedure. The project was interested in 
the null hypothesis that there would be no significant difference in perform- 
ance levels between cadets at the Golden West Police Academy (the experimental 
group) and cadets at the Los Angeles Police Academy (the control group) a3 
measured by a common examination. Finding a statistically significant differ- 
ence would give cause to reject the null hypothesis , concluding that the CAI 
learning procedures were either more or less effective than the conventional 
procedures, depending upon the sign of the difference. 

Comparison of performance scores between the control and experimental 
groups with respect to the CALCOP examination enactments as well as on the 
California Short Form Test of Mental Maturity and the Wonderlic Personnel Test 
made use of the t test for significant differences in mean scores and the 
Wilcoxon matched pairs signed-rank test (Ferguson, pages 167-174). 

Examination Preparation 

Inasmuch as the purpose of the evaluative phase of Project CALCOP was to 
measure the relative effectiveness of the computer assisted instruction tech- 
niques with conventional classroom presentation techniques, a first important 
task of the project was to develop the final examination. An initial exam- 
ination was developed by instructors of the Golden West College Law Enforce- 
ment Program. The examination was tested at the Los Angeles Police Academy. 
Groups of cadets at the Academy would take the examination. After scoring. 
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the cadets and the Instructor would critique the examination in terms of 
clarity and legal accuracy. After making appropriate modifications, the 
instructor would administer the examination to a fresh group of cadets and 
repeat the evaluation. In this manner, cadet reactions to and performance 
on the examination was carefully considered in subsequent revisions of the 
final examination. Revisions were retested as described above until the final 
draft of the examination was completed. 

Control and Experimental Group Selection 

The experimental group for this study consisted initially of twenty- 
seven police cadets enrolled in the Golden 'West College/Police Academy during 
the Fall semester, 1970-71. This group undertook to study matters of search 
and seizure through independent use of the syllabus and through the use of 
the computer assisted instruction simulation exercises described earlier in 
this report . 

The control group for the experiment consisted of police cadets at the 
Los Angeles Police Academy who undertook to study matters of search and 
seizure through conventional classroom instruction as conducted at that 
Academy. Sixty police cadets out oiTa^ssS of seventy-one at the Los Angeles 
Academy took the final examination enactments. 

Members of both the control and the experimental groups took the Calif- 
ornia Short Form Test of Mental Maturity and the Wonderlic Personnel Teste 
Using the IQ scores achieved on the California Short Form Test of Mental 
Maturity as a basis, twenty-seven members of the Los Angeles Police Academy 
group were selected so as to give twenty-seven matching pairs of cadets, one 
group each from the Golden West College Police Academy and from the Los 
Angeles Police Academy. 
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The District*, employed a number of tests and comparisons making use of 
Wonderlic and IQ scores. As a result of these calculations and comparisons., 
iw made the following observations: 

1. There is no significant difference in mean IQ scores as 
measured by the California Short Form Test of Mental 
Maturity between the trrenty- three member experimental 
group at Golden West College and the tt*enty-three member 
control group at the Loa Angeles Police Academy. 

2. There is no significant difference In mean Wonderlic 
Personnel Test scores between the control group and the 
experimental group. 

3. The control group of Los Angeles Police Academy cadets 
is a representative sample In terras of IQ and Wonderlic 
scores of the total seventy-one member group of Los Angeles 
Police Academy cadets. 

4. There Is no significant difference in either mean IQ scores 
or In mean Wonderlic scores between the Golden West College 
experimental group and the total group of Los Angeles Police 
Academy cadets . 

Accordingly, any differences to be found between performance levels on 
the examination enactments as between Group 1 and Group ?. could not be 
attributed to differences in ability as measured by the California Short Form 
Test of Mental Maturity and the Wonderlic Personnel Test. Differences In 
performance levels on the final examination must be accounted for by other 
factors. 



Training 

Police cadets at the Los Angeles Police Academy studied materials relating 
to proper procedures In search and seizure masters under conventional clcss-^ 
room instruction. This instruction consisted of lectures and classroom dis- 
cussions. Police cadets at the Golden West College Police Academy studied 
the same materials making use of the study syllabus and the computer assisted 
simulated case problems. This group received no classroom instruction. 
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Examining 

After completing the training program in search and seizure, cadets at 
both the Police Academy in Los Angeles and the Academy at Golden West College 
completed a written examination consisting of case problems or enactments in 
which the student was asked specific questions about procedures and matters 
of fact relating to the situation described. 

Results 

Both the t test and the Wilcoxon matched pairs signed-rank tests were 
applied to the performance scores on the examination enactment. In every 
case, cadets at the Golden West Police Academy performed better on the final 
examination than did cadets at the Los Angeles Police Academy and in every 
case the difference In performance levels was statistically significant at 
the .05 level of confidence. 

Conclusions 

The most obvious conclusion to be drawn from the procedures outlined 
above says that the learning procedures followed at Golden West College In 
the area of search and seizure were more effective than were the procedures 
followed at the Los Angeles Police Academy, at lease as measured by the final 
examination. The District has this to say in their final report on Frojeet 
CALCOP. 

We assert, and our conclusions here are based upon 
the statistics reported above, that this basic difference 
In Instructional approach accounts for the differences 
we find In performance levels between the Los Angeles 
Police Academy conrol group and the Golden West College 
Academy experimental group (Project CALCOP Final Report . 
p. 23). 
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Following this assertion, the final report considers a number of other 
factors that might account for the performance differences between the con- 
trol and experimental groups. These factors Included experimental bias 
(the so-called Hawthorne effect), examination procedures and previous experi- 
ence of police cadets (Project CALCOP Final Report s pp. 23-27). Dispatching 
these as matters not affecting cadet performance through the use of statistical 
analysis, the investigators summarized : 

Although we are not prepared on the b^sis of Project 
CALCOP to conclude that the computer assisted learning 
portion of the learning system devised is more effective 
than classroom instruction, we do think that the total 
learning system including independent study of the syllabus 
as well as computer assisted case problems, presents a 
more effective learning environment in the area of search 
and seizure than does conventional classroom instruction. 

This is not to say, of° course, that conventional class- 
room instruction has been other than excellent in 
quality* In fact we cannot say, as a result of this 
study, that it has been good, bad, or indifferent. 

Rather , we have found evidence that instructional 
effectiveness in search and seizure can be further 
improved through the use of learning systems similar to 
that developed by project CALCOP (Project CALCOP Final 
Report , pp. 27-28). 



INFORMAL EVALUATION 

Most instructional evaluation at the Coast Community College District 
takes place on an informal basis and draws heavily on student reaction to 
the total Instructional program. Student reaction Is often solicited through 
the use of end-of-the course questionnaires administered to students by faculty 
members. Quite often, even more casual observation serves as the basis of 
evaluating the effectiveness of the particular type of instruction. 

This portion of the paper presents some verbatim comments delivered to 
the Office of Educational Development by faculty members who were engaged in 
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evaluating instructional systems prepared under the auspices of the Faculty 
Fellowship Program. The first of these reports the degree to which students 
made some use of sound-on-slide materials developed as a supplementary learn- 
ing system for large group instruction in Psychology 1A. The report identifies 
a couple of problems in conducting large group instruction and then presents 
the sound-on-slide system as an effort to solve some of these problems. The 
second report , and the more rigorous of the three, presents the findings of a 
fairly extensive survey of students making use of the Math-Tutorial Center at 
Golden West College. In addition to soliciting student views on it, some 
effort was expended toward a normative evaluation of actual student performance. 
The third report, making only casual reference to student performance, devotes 
itself mostly to evaluating the mechanical devices used in the project and to 
clapping the Office of Educational Development and other District workers on 
the back for their help in executing the project. 



3ound-0n-Sl id es 

This project was to develop "100 sound-on-slide conceptual units to be 
used by the division in it’s instructional activities. 11 It was understood 
that this objective was so designed as to allow a high degree of flexibility 
as needs become evident during the development of the project. 

Procedures - After purchasing the necessary supplies and equipment, dis- 
cussing and demonstrating the hardware capabilities with members of the staff 
and beginning a unit on the history of Psychology; 3ome considerations that 
were to guide the rest of the project emerged. In effect they were as follows: 

1. Certain students in Introductory Psychology find the large-group 
situation both frustrating and difficult. Often the core of the 
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dissatisfaction stems from the perceived difficulty of the evalu- 
ation instruments used in the class. Though a great deal of effort 
had been made to standardize test feedback, this aspect of the 
instructional program left much to be desired. 

2. Unfortunately, the large-group situation often seemed most discon- 
certing to the highly motivated student and to the student who was 
slightly below average in learning ability. It was felt that both 
these groups of students could profit from some form of individualized 
access to test feedback. 

In a converstaion with Roy Andreen, (Dean, Counseling and Guidance) he 
suggested the possibility of using the sound-oa-slide equipment for test feed- 
back. This set the wheels in motion for what appears to be a very effective 
innovation In a Psychology program. 

During the Spring semester of 1970, each question used In the large-group 
was typed on an 8 1/2 11 X 11” sheet of plain paper with the primary typewriter. 

A 35mm slide was made of them and they were placed in the sound-on-slide 
machine. The teacher who had submitted each question then prepared a 35 second 
audio explanation explaining the principle, concept of objective of the ques- 
tion, the correct answer, why that was the correct answer, and where the answer 
could be found (in the lecture or text page). After each 50-question examina- 
tion these units were made available In the multi-media center In the library 
on a voluntary basis to students. 

It Is significant that the final examination in this course is compre- 
hensive and consists of 100 questions. Fifty of these questions reflect the 
principles already tested for during the semester. The assumption Is that a 
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motivated student can go over the 150 questions in the machine just prior to 
the final and have reviewed for him approximately one-half of the final. 

We demonstrated this equipment to the students in the large-group imme- 
diately after the first examination and instructed them regarding its avail- 
ability. 

Findings - Vfereceived considerable positive feedback from both students 
and seminar teachers. The most significant consideration 3eems to me to be 
the number of students who availed themselves of the equipment. According to 
the console operator, 371 requests were logged for the machine. In view of 
the fact that there were usually several students watching it at once — on 
occasion as many as ten — it becomes evident that the students did perceive it 
as an effective adjunct to their study. 

Conclusions - We are involved right now in preparing the test questions 
for our first fifty question examination. It is a time consuming, at times, 
laborious task but it appears that the staff considers it worthwhile. I sus- 
pect that we have, with a very moderate investment, offset some of the dis- 
advantages of the large-group class. Hopefully, that student who doesn’t 
learn too rapidly, and the one who is willing to spend extra time to do better 
now has at his disposal a structured review with some answers as well as soma 
reasons why, pertinent to the objectives of the course. 

Someday, when time permits, I would still like to put together a few 
units on some topics in Psychology. The slides on a unit in the History of 
Psychology are already finished. This, however- will have to wait because 
we do feel that existing use of our time, and that of the machine, is being 
used most effectively in the present application. 
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Math Tutorial Center 

As the result of funding through a Faculty Fellowship, the Golden West 
Math Council was able to offer a mathematics tutorial service during the 
Spring semester of 1970* This service was available to students seeking 
individual assistance in mathematics. 

The purpose of this paper is to describe the tutorial services provided, 
to report on the data collected regarding the services, and to offer an 
evaluation and recommendations which may be useful for the operation of future 
mathematics tutorial services. 



Description of the Services - 



Purpose: A Mathematics Tutorial Center was established at Golden 

West to provide individual assistance to students having 
difficulty in mathematics. 



Facilities: A room in the College Center v;as reserved for the tutoring 

services* The room was equipped with tables, chairs and 
a portable blackboard. 

Hours: The services began on February 11 and continued until the 

beginning of final examination (June 4) and was available 
on a daily basis between the hours of 8 a.m. and 2 p.m. 

Staff: Members of the mathematics faculty and nine advanced math 

students served as tutors for the Center with from one to 
three tutors assigned each hour in the Center. The stu- 
dent tutors hired were recommended by the math faculty. 

Each math instructor donated two hours a week in the 
Center while the student tutors spent a total of 40 hours 
a week in the Center, making the totpl tutor time 52 hours. 

The Math/Science Division Counselor acted as Director of 
the Tutorial Center and a student lab assistant was hired 
to assist with the clerical work. 

r v -search and Data Collected - Throughout the semester several procedures 
were used to collect data and provide research relevant to the tutoring ser- 
vices; 
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a. “Log" sheets were maintained on a daily basis in the Center to 
record the studenV use of the services. 

b. Questionnaires were distributed in all the math classes to*rard the 
end of the semester primarily for the purpose of evaluating the 
services. 

c. Analysis of previous math grades and Spring semester grades was 
done in an attempt to determine if there was a difference in per- 
formance in math by those students who used the services as com- 
pared with students who did not use the services. 

Following is a summary of what appears to this writer to be significant 
information provided by the data collected from the "log" sheets and question- 
naires. 



Who Used the Tutorial Services? 

a. Of the 173 students who used the services, 86% were pre-calculus 
students . 

b. Students from Math A, C, and D accounted for 60% of the total 
number of students who used the services. 

c. At least 35% of the students who registered in a math class 
Spring semester used the tutorial services. 

How Often Were the Tutorial Services Use d? 

a. According to the "log” sheets, at least 173 students used the Center 
a total of 876 times. 

b. The Center was used by pre-calculus students 85% of the time, with 
students from Math A, C, and D using the services 60% of the time. 
(There is evidence that not all students signed the "log" sheet 
each time they used the Center. It is probable that the actual 
number of times they spent in the Center is greater than 876.) 

c. The responses on the questionnaires indicate 44% of the students 
who used the services returned 5 or more times. 

What Reasons Did the Students Give for Not Using the Center? 

a. "I did not need any tutoring in math/* was the reason given most 
often for not using the Center (1/3 of the responses). 

b. Twenty percent said they did not use the Center because the times 
available for tutoring were not convenient for them, with an equal 
number of students selecting “other* 1 as the reason for not using 
the Center. 

What Reasons Did the Students Give for Not Returning to the Center ? 

a. The reason given 30% of the time was that they did not need any 
tutoring. 
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b. Seventeen students said they were not satisfied with the help 
the received when they did go. 

c. Twelve students said the hours available were not convenient 
to them. 

When Was the Tutorial Center Used Moat Often? 

a. The distribution of students using the Center per day was fairly 
even, with a slightly higher attendance on Wednesday and Thursdays, 
and the lowest percentage of attendance on Mondays. 

b. The distribution of students using the Center "T hour was 
heaviest during the morning hours, from 8 to jlx a.m,* gradually 
tapering off from 11 to 2. 

During What Hours Would the Students Prefer the Tutorial Center to be 
Available? 

a. Eighty percent of the response indicated a preference for the 
Center to be open between 7 a.m. and 5 p*m. with 75 % favoring 
the hours between 9 a.m. and 3 p.m. 

b# Fourty students expressed a desire for the Canter to be open in 
the evenings# 

How Did Students Rate the Help Given at the Center? 

a. In rating the student tutors, 65% of the pre-calculus students 
said all or most of the student tutors were very helpful. 75% 
of the calculus students said all or most of the student tutors 
were very helpful. 

b. In rating math instructors, 85% of pre-calculus students said 
most or all of the math Instructors were very helpful. 84% of 
the calculus students also said most of the instructors were 
very helpful. 

What Effect Did Students Feel the Tutorial Center Had on Their Attitudes, 
Understanding, and/or Grades in Math? 

a. Over 50% of the students said their attitude about mathematics had 
improved since they had been at Golden West, 40% said their attitude 
had stayed the same, and 6% said their attitude about math had 
become worse. 

b. The students attributed their change on attitude most often to their 
math instructors at Golden West (66% of the responses). 

c. Sixty percent of the students who used the tutorial services attri- 
buted their change to their instructors, while 18% attributed the 
change to their use of the Tutorial Center. 

d. 1/4 of the students felt their use of the Center had resulted In 
their getting better grades in math. 

e# Sixty percent of the students who used the Tutorial Center felt 

their use of the services had increased their understanding of math. 

f. Le9s than 2 % said it confused them more than helped them. 
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g. Fourteen percent said their use of the Center made no difference 
in their performance in math. 

How Necessary Did the Students Feel the Center Was and How Did They Feel 
It Compared To Other Sources of Help ? 

a. Seventy-five percent of the students felt they would have received 
the help they needed if there had been no Tutorial Center. 

b. Of those students who said they would gotten (sic) the help they 
needed, 42% indicated they would have gotten it from their math 
instructor while 47% said they would have gotten help from another 
student or friend. 

c. Twenty percent felt this help would have been better than the 
Center, 56% felt it would have been as adequate as the Center, and 
24% felt the help would not have been as adequate as the Tutorial 
Center . 

What Criticism Was Given Regarding the Center ? - Following are comments 
given most often by students as paraphrased by this writer: 

a. Students disliked having to wait for help during the busy hours. 

b. They expressed a need for more tutors and quieter facilities. 

c. Many felt the Center’s hours were not convenient. 

d. Higher math students expressed a need for more advanced tutors. 

What Did the Students Like About the Center ? - Following are comments 
given most often by students as paraphrased by this writer: 

a. A number of students indicated the tutors were always willing to 
help and were very patient* 

b* Some students said they liked the Center because they felt more 
comfortable getting help from students than instructors. 

c. Many students felt it lessened anxiety as it was "always there 
when you needed help*" 

d. Students indicated they were able to get more individual attention 
than was possible in class „ 

e= A common conanent was "it made the difference." 

Probably the most important question to be answered with regards to the 
mathematics tutorial services is did it make a difference in how a stud ect 
performed in mathematics? As a result of using the services, did he under- 
stand math better or get better grades or did it keep him from dropping; Ms 
math class? According to the questionnaires, the students felt using the 
tutoring services made a difference or did these things. 
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An attempt wag made to determine if there was* in fact, a difference in 
the performance in mathematics between those students who used the services 
and those who did not use the services. From hereon.* those students who 
used the services will be known as Group 1 and students who did not use the 
services as Group 2. Group 1 was arbitrarily defined as any day school math 
student at Golden West who used the tutorial services 5 or more times while 
Group 2 was determined as any student enrolled in a day school math class 
(excluding Math 51, A(R), D(R) , 7, 42, 44) at Golden West who used the ser- 
vices less than 5 times as recorded on the "log" sheets. 

Several comparisons were made between Group 1 and 2 to determine if 
there was a significant difference in the performance of the two groups. 
Before the results of these comparisons are given, some of the limitations 
of the methods used in testing for a significant difference should be pointed 
out: 

1. No attempt was made to match students who used the services with a con- 
trol group; all students who wanted to use the tutorial services were 
allowed to do so. One may therefore conjecture that Group 1 is a biased 
sample simply because of their motivation to use the tutoring services 
as compared to Group 2 who chose not to use the services. It is prob- 
ably true Group 1 is a different type of student in the first place and 
unrepresentative of the population, thus any difference in math perform- 
ance cannot necessarily be attributed to the use of the math tutorial 
services. 

2. The statistical techniques used for tests of significance had to assume 
the groups were randomised with regards to instructors, math courses and 
grading policies. The number of students involved was too small to keep 
these variables constant for the comparison of any particular group. 

Following is a summary of the comparisons made between Groups 1 and 2 
and the results of the comparisons. 

1. Was there a significant difference between the groups in the performance 
in their previous math class? Were the students who used the tutoring 
services better or worse students in math to begin with than the students 
who did not use the services? 
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A comparison of the grade point averages of Group 1 and Group 2 in their 
last math class showed no significant difference between the groups in 
their previous grades in math classes* 

2. Was there a significant difference in the grade point averages of the 
two groups as determined by their final grades in math Spring semester? 

Did the students who used the services do better or worse in math Spring 
semester than those who did not use the services? 

A comparison of the grade point averages of Group 1 and Group 2 in their 
final grades Spring semester in math showed no significant difference 
between the groups in their performance in mathematics. 

3. Was there a significant difference between the groups when their Spring 
semester final grades were compared with their grades in their previous 
math class? That is, did students who used the services do better on 
the average, or worse in their present math class than they had done pre- 
viously in math compared with those who did not use the services? 

i? 

l s 

The difference between the grade point average of the previous math course 
and the grade point average of the Spring semester math course was deter- 
mined from Groups 1 and 2. A comparison of this grade point average 
difference showed no significant difference between the groups. 

Evaluation and Recommendations - From the total number of student contact/ 



hours in the Center and the effect the student said the tutorial services 
had on their attitudes, understanding and grades in math, one can conclude 
the Math Tutorial Center was very successful in fulfilling a valuable need 
for a substantial number of students* In summarizing the evaluations given by 
students and instructors regarding the Center, greatest benefits of the ser- 
vices seemed to be from the fact chat students had the opportunity to receive 
one-to-one assistance from qualified individuals at the time they needed it. 

With the exception of a need for more tutors at peak times during the 
semester, the facilities, staff and method of operation of the Tutorial Center 
appeared to be more than adequate for the successful functioning of the Center. 
The following changes are offered for improvement in the operation of future 
math tutorial services. 
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1> More tutor time be concentrated at the beginning of the semester, pri- 
marily in the morning hours. 

2. A sufficient number of tutors be hired who are particularly qualified in 
tutoring the subjects of elementary algebra, intermediate algebra, and 
trigonometry. 

3. Student tutors be oriented in how to tutor students. For exarole, not to 
spend too much time with one student while others are waiting; not to do 
all the student’s homework? if can’t answer question, refer student to an 
instructor. 

4. Consideration be given to offering math tutorial services in the evening. 

5. Post the times there will be tutors in the Center who are particularly 
qualified to assist higher math students. 



Producing Media For Medical-Surgical Nursing 

This project was prepared to utilize the audio-tutorial laboratory for 
individual study, to produce a syllabus for Nursing 30 (Medical- Surgical-I) 
and to prepare some slides-tape programs on selected subjects. Two of these 
objectives have been completed. 

P reparation £ Findings - 

1. A 175 page syllabus for Medical-Surgical Nursing I was produced a.Kt 
will be ready for student purchase in the bookstore for Fall semester. 

2. Approximately 200 slides were made on the subjects of water and 
electrolyte balance in relation to the patient with gastrointestinal, 
kidtiey, and lung problems. Audio-tapes, worksheets, and post- tests 
were also prepared for these exercises. The results of the post-test 
indicate that the majority of the students who did all the exercises 
in the audio-tutorial laboratory scored higher than students who did 
only the assigned work. 

A questionnaire on various teaching method preferences was given 
to the freshman nursing students. The results indicated a decided 
preference for classes having direct teacher involvement and guid- 
ance such as lectures, guest speakers, movies and SAS sessions. The 
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majority felt that the audio-tutorial sessions were helpful and 
should be part of the course requirements. Half of the group pre- 
ferred small groups for viewing the filmstrips and slides x*hile the 
other half stated a preference for individual viewing and study. 

3. The third objective of utilizing the audio-tutorial laboratory for 
individual study was somewhat limited because not all the equipment 
for individual students use has not arrived. However, we were able 
to evaluate the Graf lex Rear-View Slide Projector for individual use. 
Gur conclusions are that the machine is far from student-proof. Each 
time the machine was used the slides would jam, making it impossible 
to complete the exercise. Also, the slides slip readily out of the 
carousel when being handled. 

The Dukane Filmstrip A-V-Matie projectors have not yet arrived 
so that we were unable to evaluate them in relation to individual 
student learning. We have one which was used for small groups of 
students. The A-V-Matic appears to be easy for the student to work 
and handle. Also, it can be placed in the individual carrels in the 
audio-tutorial lab. 

Another aspect of the project was to review commercially prepared media 
for possible purchase. A number of filmloops, filmstrips, movies, etc., were 
provided. Our recommendations will be submitted to the Division. 

Conclusion - Just a personal comment - anyone wishing to do anything 
"innovative" must be prepared to meet up with numerous little frustrations 



along the way. The problems with how to do this and that, and all the red- 
tape could discourage one. However, even though the results of what has b sen 
done are inconclusive on the project, I found that the people who are our 



31 



backbone - the IHC and multi-media staff, my student typist, the artist doing 
the drawings, the staff (in particularly the vocational office) bailed me out 
many time's. The student reaction, too, has been just great, so it’s been 
frustrating but fulfilling. It’s a real learning experience for this teacher. 

CONCLUSIONS 

In general, efforts at evaluating instructional effectiveness are either 
normative in nature or solicit student opinions as to the quality of the 
performance. In order for Instructional evaluation to be performed most 
meaningfully. It must be conducted In terms of specific educational objectives 
that are being taught by the educational systems under question. In addition, 
some rigorous efforts must be devoted to selecting control and experimental 
groups and in pre- and post-testing the students undergoing the training pro-^ 
gram. Without these constraints 9 no effective evaluation can take place. 
Unfortunately, this type of thing Is very difficult to perform. By and large, 
teachers are unwilling to have their students treated as guinea pigs and react 
rather negatively to the whole notion of evaluation efforts conducted in a 
rigorous, quantifiable manner. "After all," many argue, "the things I teach, 
and the people I try to teach it to, cannot be measured in quantified terms. 

How can you evaluate the quality of my efforts by assigning numbers to these 
things?" 

More specifically, the difficulties in evaluating instructional effec- 
tiveness are four in number. First, and this has been mentioned already, 
teacher unwillingness to use their students as guinea pigs is the most trouble- 
some problem. Second, conducting comparisons that are, in fact, fair and 
selecting experimental and control groups that are, in fact, comparable is a 
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most difficult job. Three, establishing criteria of success or criteria of 
evaluation is also most difficult, Four, the validity of measuring instru- 
ments for educational programs is also sometimes difficult to establish. By 
and large, all four of these difficulties combine into a total situation that 
makes the researcher's job most difficult. 

It would have been best if this paper could have compared relative costs 
of a particular learning system and the relative effectiveness of that system 
as well with conventional instruction. This could not be done. Those few 
examples in which rigorous analysis of comparative effectiveness are avail- 
able are such that rigorous cost analysis cannot be performed. The summative 
examples of effectiveness comparison presented here both involved the use of 
computer assisted instruction. To this writer's knowledge this is the only 
serious effort at evaluating two instructional systems based upon specifically 
stated educational objectives available in the District. As both of these 
systems make use of computer assisted instruction it remains then to evaluate 
these systems in terms of the relative costs of computer assisted instruction 
as compared to conventional instruction. As the computer system is utilised 
in the Coast Community College District, however, there is nc adequate way to 
calculate the cost involved in conducting computer assisted i!istruction. Even 
if there were, no records are maintained as to the amount of time any one stu~ 
dent spends at a computer terminal undergoing computer assisted instruction. 
Thus, even if we could determine a dollar figure per hour of computer assisted 
instruction, we have no way of knowing how many hours any one student spent 
making use of it. Once again, it's a cost accounting problem, complicated by 
the problem of keeping track of who is making use of what terminals. The 
District has been loath to formalize the manner in which students make use of 
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computer terminals, feeling that sign-in sheets and other check-in, check-out 
procedures would tend to dissuade students from participating in this activity 
The lack of control that results is probably worth it. 
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